Papers with language modeling objective
Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers (2025.tacl-1)
Copied to clipboard
Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar, Tianxing He, Noah A. Smith, Navin Goyal, Yulia Tsvetkov
| Challenge: | Inductive biases in transformers can cause hierarchical generalization without explicitly encoding structural bias. |
| Approach: | They investigate sources of inductive bias in transformer models and their training that could cause such preference for hierarchical generalization. |
| Outcome: | The proposed model can generalize to novel syntactic forms without explicit bias . the proposed model is able to generalize on a dataset with a hierarchical grammar . |
sudoLLM: On Multi-role Alignment of Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a framework that allows users to control access rights has not been extensively studied in the large language model realm. |
| Approach: | They propose a framework that allows users to control access rights in a multi-role manner. |
| Outcome: | The proposed framework improves alignment, generalization and resistance to prefix-based jailbreaking attacks. |
Intent Discovery for Enterprise Virtual Assistants: Applications of Utterance Embedding and Clustering to Intent Mining (2022.naacl-industry)
Copied to clipboard
| Challenge: | Existing approaches to clustering unlabeled utterances are based on transformerbased sentence embedding methods. |
| Approach: | They propose a semantic embedding approach that can be leveraged to identify clusters of utterances that correspond to unhandled intents. |
| Outcome: | The proposed approach can identify clusters of utterances that correspond to unhandled intents from a large collection of enterprise virtual assistant data using a multi-task softmax loss. |
More room for language: Investigating the effect of retrieval on language models (2024.naacl-short)
Copied to clipboard
| Challenge: | Retrieval-augmented language models are a promising alternative to standard pretraining, but little attention has been put into understanding what this type of training scheme does to the underlying language model when analyzed as a standalone -separated from the overall retrieval pipeline. |
| Approach: | They propose an ‘ideal retrieval’ methodology to study these models in a fully controllable setting and propose a retrieval augmentation methodology to examine their effects. |
| Outcome: | The proposed model saves substantially less world knowledge in their weights, but is worse at comprehending global context. |
Answer-level Calibration for Free-form Multiple Choice Question Answering (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models exhibit few-shot and zero-shot learning capability on tasks such as commonsense reasoning. |
| Approach: | They propose to model context-independent biases in terms of the probability of a choice without the context and to remove it using an unsupervised estimate of similarity with the full context. |
| Outcome: | The proposed model improves over baselines on commonsense reasoning tasks. |
Modeling Code-Switch Languages Using Bilingual Parallel Corpus (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for bilingual language modeling are limited due to lack of training data and syntactic structure. |
| Approach: | They propose a bilingual attention language model that performs language modeling objective with a quasi-translation objective to model the monolingual and cross-lingual sequential dependency. |
| Outcome: | The proposed model reduces the perplexity of 20.5% over the best-reported model. |
Understanding by Understanding Not: Modeling Negation in Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Negation is a core construction in natural language, but state-of-the-art pre-trained language models often handle it incorrectly. |
| Approach: | They propose to augment language modeling objective with unlikelihood objective based on negated generic sentences from a raw text corpus. |
| Outcome: | The proposed approach reduces the top 1 error rate to 4% on negated LAMA dataset and improves on negating NLI benchmarks. |
MAGMA – Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pretraining is becoming the norm in Vision-Language (VL) modeling. |
| Approach: | They propose a method for augmenting generative language models with additional modalities using adapter-based finetuning. |
| Outcome: | The proposed method outperforms Frozen on open-ended generative tasks while maintaining the language model weights. |
Pre-Training to Learn in Context (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are not explicitly trained to learn in context. |
| Approach: | They propose a framework to enhance in-context learning by pre-training language models on a large collection of "intrinsic tasks" they evaluate the in-constitution learning performance of the model trained with PICL on seven widely-used text classification datasets and the Super-NaturalInstrctions benchmark . |
| Outcome: | The proposed framework outperforms larger language models with nearly 4x parameters on seven widely-used datasets and the Super-NaturalInstrctions benchmark. |
Improving Transformer Models by Reordering their Sublayers (2020.acl-main)
Copied to clipboard
| Challenge: | a sandwich transformer pattern is a new approach to multilayer transformers that can be used for different tasks. |
| Approach: | They propose a transformer ordering pattern that reorders sublayers in a sandwich transformer pattern . they generate random transformer models and train them with the language modeling objective . |
| Outcome: | The proposed pattern improves perplexity on multiple word-level and character-level language modeling benchmarks at no cost in parameters, memory, or training time. |
On the Multilingual Capabilities of Very Large-Scale English Language Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Generative Pre-trained Transformers (GPTs) have been scaled to unprecedented sizes in the history of machine learning. |
| Approach: | They investigate the potential and limits of Generative Pre-trained Transformers in three tasks . they find it can be almost as useful for many languages as it is for English . |
| Outcome: | The proposed model can perform tasks in five different languages, and its potential is explored . it can learn from a few examples "via text interaction" and is scalable to many languages . |
Visually Grounded Compound PCFGs (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on visual groundings for language understanding has been drawing much attention. |
| Approach: | They propose to use an extension of probabilistic context-free grammar model to do fully-differentiable end-to-end visually grounded learning. |
| Outcome: | The proposed model outperforms the previous grounded model and significantly outperformed the previous model on the MSCOCO test captions. |
Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2022.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained language models struggle in tasks that require reasoning . recent work shows that they struggle in performing symbolic reasoning operations without substantial amounts of additional data. |
| Approach: | They propose to leverage semi-structured tables and generate at scale question-paragraph pairs where answering the question requires reasoning over multiple facts in the paragraph. |
| Outcome: | The proposed model outperforms T5, a popular pre-trained encoder-decoder model, on reasoning-focused reading comprehension datasets. |
FIRST: Faster Improved Listwise Reranking with Single Token Decoding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing listwise LLMs lack efficiency as they provide ranking output in the form of a generated ordered sequence of candidate passage identifiers. |
| Approach: | They propose a listwise LLM reranking approach that leverages the first generated identifier to obtain a ranked ordering of the candidates. |
| Outcome: | The proposed approach accelerates inference by 50% while maintaining robust ranking performance with gains across BEIR benchmark. |